Statistical Inference After Model Selection

Overview of the presentation

• Background & motivations
• Why is model selection an issue?
• Simulations of Model-Selection
• Summary
• Potential solutions

Background

• Classic regression analysis treats predictors as fixed or non-random.

• Allows estimation of unbiased coefficients.

Motivations

• Berk and colleagues (2009) claim Model Selection is ubiquitous in Criminology.

• When the correct model is unknown…

  • Set of Models
  • Data is examined to choose the “best” model
  • Parameters are estimated
  • Statistical inference

• However, what happens when the same dataset is used?

Why is model selection an issue?

• Sampling distributions become distorted

• Estimates are biased (e.g., Beta coefficients, standard errors)

• Overconfidence in results (inflated Type 1 error)

Why is model selection an issue?

Specific Mechanisms

• Direct Censoring

• Indirect Censoring

• Alterations in dispersion of regression parameter estimates

Simulations of Model-Selection

• Forward stepwise regression using AIC applied to 10,000 samples of \(n\) = 200

Simulations of Model-Selection

reps  <-  10000
p <- 3
Sigma <- matrix(c(5,4,5,
                  4,6,5, 
                  5,5,7), p, p)
n <- 200
betas <- c(3, 0, 1, 2)

rsq <- NULL
coefs <- cover <- matrix(NA, nrow = reps, ncol = 3)
colnames(coefs) <- c("w", "x", "z")
colnames(cover) <- c("w", "x", "z")

Simulations of Model-Selection

reps  <-  10000
p <- 3
Sigma <- matrix(c(5,4,5,
                  4,6,5, 
                  5,5,7), p, p)
n <- 200
betas <- c(3, 0, 1, 2)

rsq <- NULL
coefs <- cover <- matrix(NA, nrow = reps, ncol = 3)
colnames(coefs) <- c("w", "x", "z")
colnames(cover) <- c("w", "x", "z")

for (i in seq(reps)) {
  X <-  MASS::mvrnorm(n = n, rep(0, 3) , Sigma)
  y <- as.numeric(cbind(1, X) %*% betas + rnorm(n, 0, 10))
  Xy <- as.data.frame( cbind(X, y))
  colnames(Xy) <- c(c("w", "x", "z"), "y")
  fit <- lm(y ~., data = Xy)
  sel <- step(fit, k = 2, trace = FALSE)
  s <- summary(sel)
  tvals <- s$coefficients[,3][-1]
  coefs[i, names(tvals)] <-  tvals
  rsq[i] <- s$r.squared
}

Distribution of models selected in 10,000 draws

• The \(R^2\)s varied over the simulations between about .3 and .4.

• For \(X\) the post-model selection distribution has a greater mean (2.6–2.2) and a smaller standard deviation (.79–1.0).

• For \(Z\) the mean and the standard deviation are biased substantially upward: from 4.9 to 5.5 for the mean and from 1.0 to 2.3 for the standard deviation.

Statistics (our results)

t-values for \(\beta_2\)

• Red curve/ solid line = Conditional on preferred model being known

• Blue curve/ dashed line = Conditional on predictor being included in a model

t-values for \(\beta_3\)

• Red curve/ solid line = Conditional on preferred model being known

• Blue curve/ dashed line = Conditional on predictor being included in a model

False power

• For the preferred model, power to reject \(H_0: \beta_2=0\) with \(\alpha =0.05\) is approximately 60%.

• After model selection, that probability is about 76%.

• Bias due to model selection artificially inflates power in about 27%.

Selecting the preferred model (\(X\) and \(Z\) included) does not guarantee any of the desirable properties of the regression coefficient estimates.

Summary

• Post-model-selection statistical inference can lead to biased regression parameter estimates and seriously misleading statistical tests and confidence intervals.

• The particular selection procedure used does not materially matter.

• Sometimes the correct sampling distribution and the post-model-selection sampling distribution will be very similar.

Potential solutions

Discussion